Search Results: "craig"

30 December 2015

Craig Small: Forking processes and Gtk2

I made a change recently on the gjay program. gjay is a gtk program that basically analyzes your music and makes playlists. There is a gui frontend and a analyzer back-end and they communicate through a pipe. One really useful debugging option gtk has is to make warnings fatal, so when gtk finds one it crashes at that point and you can use gdb to trap it. The flag is g-fatal-warnings. I have been updating gjay and initially it didn t have this option, so I needed to add the gtk options, which is a simple one-liner. But then gjay gave some odd errors about XCB. Often, but not every time gjay started on stderr the following cryptic messages appear:

[xcb] Unknown sequence number while processing queue
[xcb] Most likely this is a multi-threaded client and XInitThreads has not been called
[xcb] Aborting, sorry about that.
forktest: ../../src/xcb_io.c:274: poll_for_event: Assertion  !xcb_xlib_threads_sequence_lost' failed.

OK, so part of my init sequence was unhappy. The odd thing was it only appeared when I added the gtk options. I narrowed it down with some test code which displayed the error but stripped all the other parts out.

```
#include <gtk/gtk.h>
```
```
 
```
```
int main(int argc, char *argv[])
```
```
 
```
```
    GOptionContext *context;
```
```
    GError *error;
```
```
    pid_t pid;
```
```
    GtkWidget *win;
```
```
 
```

    context = g_option_context_new("My test");

    g_option_context_add_group (context, gtk_get_option_group (TRUE));

```
    error = NULL;
```

    if (!g_option_context_parse(context, &argc, &argv, &error))

```
        return 1;
```
```
    pid = fork();
```
```
    if (pid < 0) 
```
```
        return 1;
```
```
    if (pid == 0)   //child
```
```
        GMainLoop *loop;
```
```
 
```
```
        loop = g_main_new(FALSE);
```
```
        g_main_run(loop);
```
```
        return 0;
```
```
      else   // parent
```

        if (gtk_init_check(&argc, &argv) == FALSE)

```
            return 1;
```

        win = gtk_window_new(GTK_WINDOW_TOPLEVEL);

```
        gtk_widget_show(win);
```
```
        gtk_main();
```
```
     
```
```
    return 0;
```
```
 
```

What we got going on here is a simple gtk program. Line 11 with the gtk_get_option_group() was the line I added. The problem is that the child process is not quite setup and when the parent goes into gtk_main then you get those xcb errors. The options need to be parsed before the fork, because one of the options is to not fork (it just runs the child code directly and uses stdout with no GUI). Gjay is obviously a lot more complex than this, but follows the same pattern. There is the front-end looping through gtk_main and the back-end looping on g_main_run. The backend needs to use glib not gtk as there is a non-gui option which can be used. The solution is actually (but not obviously to me) in the documentation for gtk_get_option_group. The parameter for that function whether to open the default display when parsing the commandline arguments . Changing the TRUE to FALSE on line 11 stops XCB from complaining! The screen still appears fine but not more strange XCB messages.

14 November 2015

Craig Small: Mixing pysnmp and stdin

Depending on the application, sometimes you want to have some socket operations going (such as loading a website) and have stdin being read. There are plenty of examples for this in python which usually boil down to making stdin behave like a socket and mixing it into the list of sockets select() cares about. A while ago I asked an email list could I have pysnmp use a different socket map so I could add my own sockets in (UDP, TCP and a zmq to name a few) and the Ilya the author of pysnmp explained how pysnmp can use a foreign socket map. This sample code below is merely an mixture of Ilya s example code and the way stdin gets mixed into the fold. I have also updated to the high-level pysnmp API which explains the slight differences in the calls.

```
from time import time
```
```
import sys
```
```
import asyncore
```

from pysnmp.hlapi import asyncore as snmpAC

from pysnmp.carrier.asynsock.dispatch import AsynsockDispatcher

```
 
```
```
 
```

class CmdlineClient(asyncore.file_dispatcher):

```
    def handle_read(self):
```
```
    buf = self.recv(1024)
```
```
    print "you said  ".format(buf)
```
```
 
```
```
 
```

def myCallback(snmpEngine, sendRequestHandle, errorIndication,

               errorStatus, errorIndex, varBinds, cbCtx):

```
    print "myCallback!!"
```
```
    if errorIndication:
```
```
        print(errorIndication)
```
```
        return
```
```
    if errorStatus:
```

        print('%s at %s' % (errorStatus.prettyPrint(),

              errorIndex and varBinds[int(errorIndex)-1] or '?')

```
             )
```
```
        return
```
```
 
```
```
    for oid, val in varBinds:
```
```
    if val is None:
```
```
        print(oid.prettyPrint())
```
```
    else:
```

        print('%s = %s' % (oid.prettyPrint(), val.prettyPrint()))

```
 
```
```
sharedSocketMap =   
```

transportDispatcher = AsynsockDispatcher()

transportDispatcher.setSocketMap(sharedSocketMap)

```
snmpEngine = snmpAC.SnmpEngine()
```

snmpEngine.registerTransportDispatcher(transportDispatcher)

sharedSocketMap[sys.stdin] = CmdlineClient(sys.stdin)

```
 
```
```
snmpAC.getCmd(
```
```
    snmpEngine,
```
```
    snmpAC.CommunityData('public'),
```

    snmpAC.UdpTransportTarget(('127.0.0.1', 161)),

```
    snmpAC.ContextData(),
```
```
    snmpAC.ObjectType(
```

        snmpAC.ObjectIdentity('SNMPv2-MIB', 'sysDescr', 0)),

```
    cbFun=myCallback)
```
```
 
```
```
while True:
```

    asyncore.poll(timeout=0.5, map=sharedSocketMap)

    if transportDispatcher.jobsArePending() or transportDispatcher.transportsAreWorking():

        transportDispatcher.handleTimerTick(time())

Some interesting lines from the above code:

Lines 8-11 are the stdin class that is called (or rather its handle_read method is) when there is text available on stdin.
Line 34 is where pysnmp is told to use our socket map and not its inbuilt one
Line 37 is where we have used the socket map to say if we get input from stdin, what is the handler.
Lines 39-46 are sending a SNMP query using the high-level API
Lines 48-51 are my simple socket poller

With all this I can handle keyboard presses and network traffic, such as a simple SNMP poll.

4 November 2015

Craig Sanders: Part-time sysadmin work in Melbourne?

I m looking for a part-time Systems Administration role in Melbourne, either in a senior capacity or happy to assist an existing sysadmin or development team. I m mostly recovered from a long illness and want to get back to work, on a part-time basis (up to 3 days per week). Preferably in the City or Inner North near public transport. I can commute further if there is scope for telecommuting once I know your systems and people, and trust has been established. If you have a suitable position available or know of someone who does, please contact me by email. Why hire me? Pros:

I have over 20 years experience working with linux and unix
Over 30 years in IT, tech support, sysadmin type roles
I have excellent problem-solving skills
I have excellent English language communication skills
I can only work part time so you get a senior sysadmin at a discount part-time price, and I m able to get things done both quickly and correctly.
My programming strengths are in systems administration and automation.
I m a real-life cyborg (at least part-time, on dialysis)

Cons:

My programming weaknesses are in applications development.
I can t travel at all. I have to dialyse every 2nd night.
I can t work late (except via telecommute)
I can t drink alcohol, not even beer.
I am on the transplant waiting list. At some time in the next few years I might get a phone call from the hospital and have to drop everything with 1 or 2 hours notice and be out of action for a few weeks.

Full CV available on request. I m in the top few percent on ServerFault and Unix & Linux StackExchange sites if you want to get a preview of my problem-solving and technical communication skills, see my profile at: http://unix.stackexchange.com/users/7696/cas?tab=profile CV Summary: Systems Administrator and Programmer with extensive exposure to a wide variety of hardware and software systems. Excellent fault-diagnosis, problem-solving and system design skills. Strong technical/user support background. Ability to communicate technical concepts clearly to non-IT people. Significant IT Management and supervisory experience. Particular Skills

Unix, Linux
Internet-based Services and Security
Systems & Network Administration
Virtualisation Openstack, Libvirt, KVM, Xen, Vmware
HPC Cluster slurm, Torque, OpenMPI, pdsh
Perl, Python, shell, awk, sed, etc scripting for systems automation
Data extraction/conversion, processing, and reporting. (incl. CSV, XML, many others)
Web server administation, incl. Apache.
Web development HTML, CSS, Javascript, perl, PHP, CGI scripting etc.
Postgresql, Mysql, Microsoft SQL Server, and Oracle
DNS bind8/bind9, nlnet s nsd & unbound
SMTP postfix, sendmail, exim, qmail.
High Level Technical Support
Database design & development
Mentoring and training of colleagues

Part-time sysadmin work in Melbourne? is a post from: Errata

26 October 2015

Craig Small: ps standards and locales

I looked at two interesting issues today around the ps program in the procps project. One had a solution and the other I m puzzled about. ps User-defined Format Issue #9 was quite the puzzle. The output of ps changed depending if a different option had a hyphen before it or not. First, the expected output

$ ps p $$ -o pid=pid,comm=comm
 pid comm
31612 bash

Next, the unusual output.

$ ps -p $$ -o pid=pid,comm=comm
pid,comm=comm
 31612

The difference being the second we have -p not p. The second unexpected thing about this is, it was designed that way. Unix98 standard apparently permits this sort of craziness. To me it is a useless feature that will more likely than not confuse people. Within ps, depending on what sort of flags you start with, you use a sysv parser or a bsd parser. One of them triggered the Unix98 compatibility option while the other did not, hence the change in behavior. The next version of procps will ship with a ps that has the user-defined output format of the first example. I had a look at the latest standard, IEEE 1003.1-2013, doesn t seem to have anything like that in it. Short Month Length This one has got me stuck. A user has reported in issue #5 that when they use their locale columns such as start time get mis-aligned because their short month is longer than 3 characters. They also mention some other columns for memory etc are not long enough but that s a generic problem that is impossible to fix sensibly. OK, for the month problem the fix would be to know what the month length is and set the column width for those columns to that plus a bit more for the other fixed components, simple really. Except; how do you know for a specific locale what their short month length is? I always assumed it was three! I haven t found anything that has this information. Note, I m not looking for strlen() but some function that has the maximum length for short month names (e.g. Jan, Feb, Mar etc). This also got me thinking how safe some of those date to string functions are if you have a static buffer as the destination. It s not safe to assume they will be DD MMM YYYY because there might be more Ms. So if you know how to work out the short month name length, let me know!

15 September 2015

Clint Adams: It's just like that thing

Hi, Craige.

8 September 2015

Craig Small: WordPress 4.3 getting slow you might need an update

I recently received Debian bug report #798350 where the user had a problem with wordpress. After upgrading to version 4.3, the webservers performance degrades over time. The problem is also reported at the wordpress site with bug WordPress ticket 33423 including the fix. I have backported the relevant changeset and uploaded Debian wordpress package 4.3+dfsg-2 which only contains this changeset. For a lot of people, including myself, you probably won t hit this bug but if it impacts you, then try this update.

4 September 2015

Guido G nther: Debian work in August 2015

Debian LTS August was the fourth month I contributed to Debian LTS under the Freexian umbrella. In total I spent four hours working on:

pykerberors: Fixed a regression in DLA-265-1 affecting the default parameters of checkPassword(). This resulted in DLA-265-2.
wordpress: Craig Small updated wordpress to fix CVE-2015-2213 CVE-2015-5622 CVE-2015-5731 CVE-2015-5732 and CVE-2015-5734. I did some testing and released DLA-294-1.

Besides that I did CVE triaging of 9 CVEs to check if and how they affect oldoldstable security as part of my LTS front desk work. Debconf 15 was a great opportunity to meet some of the other LTS contributors in person and to work on some of my packages: Git-buildpackage git-buildpackage gained buildpackage-rpm based on the work by Markus Lehtonnen and merging of mock support is hopefully around the corner. Debconf had two gbp skill shares hosted by dkg and a BoF by myself. A summary is here. Integration with dgit as (discussed with Ian) looks doable and I have parts of that on my todo list as well. Among other things gbp import-orig gained a --merge-mode option so you can replace the upstream branches verbatim on your packaging branch but keep the contents of the debian/ directory. Libvirt I prepared an update for libvirt in Jessie fixing a crasher bug, QEMU error reporting. apparmor support now works out of the box in Jessie (thanks to intrigeri and Felix Geyer for that). Speaking of apparmor I learned enough at Debconf to use this now by default so we hopefully see less breackage in this area when new libvirt versions hit the archive. The bug count wen't down quiet a bit and we have a new version of virt-manager in unstable now as well. As usual I prepared the RC candidates of libvirt 1.2.19 in experimental and 1.2.19 final is now in unstable.

9 August 2015

Craig Small: procps 3.3.11

I have updated NEWS, bumped the API and tagged in git; procps version 3.3.11 is now released! This release we have fixed many bugs and made procps more robust for those odd corner cases. See the NEWS file for details. The most significant new feature in this release is the support for LXC containers in both ps and top. The source files can be found at both sourceforge and gitlab at:

My thanks to the procps co-maintainers, bug reporters and merge/patch authors. What s Next? There has been a large amount of work on the library API. This is not visible to this release as it is on a different git branch called newlib. The library is getting a complete overhaul and will look completely different to the old libproc/libprocps set. A decision hasn t been made when newlib branch will merge into master, but we will do it once we re happy the library and its API have settled. This change will be the largest change to procps library in its 20-odd year history but will mean the library will use common modern practices for libraries.

8 August 2015

Craig Small: Be careful with errno

I m getting close to releasing version 3.3.11 of procps. When it gets near that time, I generally browse again the Debian Bug Tracker for procps bugs. Bug number #733758 caught my eye. With the free command if you used the s option before the c option, the s option failed, seconds argument N failed where N was the number you typed in. The error should be for you trying to type letters for number of seconds. Seemed reasonably simple to test and simple to fix. Take me to the code The relevant code looks like this:

   case 's':
            flags  = FREE_REPEAT;
            args.repeat_interval = (1000000 * strtof(optarg, &amp;endptr));
            if (errno   optarg == endptr   (endptr &amp;&amp; *endptr))
                xerrx(EXIT_FAILURE, _("seconds argument  %s' failed"), optarg);

Seems pretty stock-standard sort of function. Use strtof() to convert the string into the float. You need to check both errno AND optarg == endptr because:

A valid but large float means errno = ERANGE
A invalid float (e.g. FOO ) means optarg == endptr

At first I thought the logic was wrong, but tracing through it was fine. I then compiled free using the upstream git source, the program worked fine with s flag with no c flag. Doing a diff between the upstream HEAD and Debian s 3.3.10 source showed nothing obvious. I then shifted the upstream git to 3.3.10 too and re-compiled. The Debian source failed, the upstream parsed the s flag fine. I ran diff, no change. I ran md5sum, the hashes matched; what is going on here? I ll set when I want The man page says in the case of under/overflow ERANGE is stored in errno . What this means is if there isn t and under/overflow then errno is NOT set to 0, but its just not set at all. This is quite useful when you have a chain of functions and you just want to know something failed, but don t care what. Most of the time, you generally would have a Have I failed? test and then check errno for why. A typical example is socket calls where anything less than 0 means failure. You check the return value first and then errno. strtof() is one of those funny ones where most people check errno directly; its simpler than checking for +/- HUGE_VAL. You can see though that there are traps. What s the difference? OK, so a simple errno=0 above the call fixes it, but why would the Debian source tree have this failure and the upstream not? Even with the same code? The difference is how they are compiled. The upstream compiles free like this:

gcc -std=gnu99 -DHAVE_CONFIG_H -I. -include ./config.h -I./include -DLOCALEDIR=\"/usr/local/share/locale\" -Iproc -g -O2 -MT free.o -MD -MP -MF .deps/free.Tpo -c -o free.o free.c
mv -f .deps/free.Tpo .deps/free.Po
/bin/bash ./libtool --tag=CC --mode=link gcc -std=gnu99 -Iproc -g -O2 ./proc/libprocps.la -o free free.o strutils.o fileutils.o -ldl
libtool: link: gcc -std=gnu99 -Iproc -g -O2 -o .libs/free free.o strutils.o fileutils.o ./proc/.libs/libprocps.so -ldl

While Debian has some hardening flags:

gcc -std=gnu99 -DHAVE_CONFIG_H -I. -include ./config.h -I./include -DLOCALEDIR=\"/usr/share/locale\" -D_FORTIFY_SOURCE=2 -Iproc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -MT free.o -MD -MP -MF .deps/free.Tpo -c -o free.o free.c
mv -f .deps/free.Tpo .deps/free.Po
/bin/bash ./libtool --tag=CC --mode=link gcc -std=gnu99 -Iproc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security ./proc/libprocps.la -Wl,-z,relro -o free free.o strutils.o fileutils.o -ldl
libtool: link: gcc -std=gnu99 -Iproc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z -Wl,relro -o .libs/free free.o strutils.o fileutils.o ./proc/.libs/libprocps.so -ldl

It s not the compiling of free itself that is doing it, but the library. Most likely something that is called before the strtof() is setting errno which this code then falls into. In fact if you run the upstream free linked to the Debian procps library it fails. Moral of the story is to set errno before the function is called if you are going to depend on it for checking if the function succeeded.

27 July 2015

Dirk Eddelbuettel: Evading the "Hadley tax": Faster Travis tests for R

Hadley is a popular figure, and rightly so as he successfully introduced many newcomers to the wonders offered by R. His approach strikes some of us old greybeards as wrong---I particularly take exception with some of his writing which frequently portrays a particular approach as both the best and only one. Real programming, I think, is often a little more nuanced and aware of tradeoffs which need to be balanced. As a book on another language once popularized: "There is more than one way to do things." But let us leave this discussion for another time. As the reach of the Hadleyverse keeps spreading, we sometimes find ourselves at the receiving end of a cost/benefit tradeoff. That is what this post is about, and it uses a very concrete case I encountered yesterday. As blogged earlier, the RcppZiggurat package was updated. I had not touched it in a year, but Brian Ripley had sent a brief and detailed note concerning something flagged by the Solaris compiler (correctly suggesting I replace fabs() with abs() on integer types). (Allow me to stray from the main story line here for a second to stress just how insane a work load he is carrying, essentially for all of us. R and the R community are so just so indebted to him for all his work---which makes the usual social media banter about him so unfortunate. But that too shall be left for another time.) Upon making the simple fix, and submitting to GitHub the usual Travis CI was triggered. And here is what I saw: first travis build in a year

All happy, all green. Previous build a year ago, most recent build yesterday, both passed. But hold on: test time went from 2:54 minutes to 7:47 minutes for an increase of almost five minutes! And I knew that I had not added any new dependencies, or altered any build options. What did happen was that among the dependencies of my package, one had decided to now also depend on ggplot2. Which leads to a chain of sixteen additional packages being loaded besides the four I depend upon---when it used to be just one. And that took five minutes as all those packages are installed from source, and some are big and take a long time to compile. There is however and easy alternative, and for that we have to praise Michael Rutter who looks after a number of things for R on Ubuntu. Among these are the R builds for Ubuntu but also the rrutter PPA as well as the c2d4u PPA. If you have not heard this alphabet soup before, a PPA is a package repository for Ubuntu where anyone (who wants to sign up) can upload (properly setup) source files which are then turned into Ubuntu binaries. With full dependency resolution and all other goodies we have come to expect from the Debian / Ubuntu universe. And Michael uses this facility with great skill and calm to provide us all with Ubuntu binaries for R itself (rebuilding what yours truly uploads into Debian), as well as a number of key packages available via the CRAN mirrors. Less know however is this "c2d4u" which stands for CRAN to Debian for Ubuntu. And this builds on something Charles Blundell once built under my mentorship in a Google Summer of Code. And Michael does a tremdous job covering well over a thousand CRAN source packages---and providing binaries for all. Which we can use for Travis! What all that means is that I could now replace the line

 - ./travis-tool.sh install_r RcppGSL rbenchmark microbenchmark highlight

which implies source builds of the four listed packages and all their dependencies with the following line implying binary installations of already built packages:

 - ./travis-tool.sh install_aptget libgsl0-dev r-cran-rcppgsl r-cran-rbenchmark r-cran-microbenchmark r-cran-highlight

In this particular case I also needed to build a binary package of my RcppGSL package as this one is not (yet) handled by Michael. I happen to have (re-)discovered the beauty of PPAs for Travis earlier this year and revitalized an older and largely dormant launchpad account I had for this PPA of mine. How to build a simple .deb package will also have to left for a future post to keep this more concise. This can be used with the existing r-travis setup---but one needs to use the older, initial variant in order to have the ability to install .deb packages. So in the .travis.yml of RcppZiggurat I just use

before_install:
## PPA for Rcpp and some other packages
- sudo add-apt-repository -y ppa:edd/misc
## r-travis by Craig Citro et al
- curl -OL http://raw.github.com/craigcitro/r-travis/master/scripts/travis-tool.sh
- chmod 755 ./travis-tool.sh
- ./travis-tool.sh bootstrap

to add my own PPA and all is good. If you do not have a PPA, or do not want to create your own packages you can still benefit from the PPAs by Michael and "mix and match" by installing from binary what is available, and from source what is not. Here we were able to use an all-binary approach, so let's see the resulting performance: latest travis build

Now we are at 1:03 to 1:15 minutes---much better. So to conclude, while the every expanding universe of R packages is fantastic for us as users, it can be seen to be placing a burden on us as developers when installing and testing. Fortunately, the packaging infrastructure built on top of Debian / Ubuntu packages can help and dramatically reduce build (and hence test) times. Learning about PPAs can be a helpful complement to learning about Travis and continued integration. So maybe now I need a new reason to blame Hadley? Well, there is always snake case ...

Follow-up: The post got some pretty immediate feedback shortly after I posted it. Craig Citro pointed out (quite correctly) that I could use r_binary_install which would also install the Ubuntu binaries based on their R packages names. Having built R/CRAN packages for Debian for so long, I am simply more used to the r-cran-* notations, and I think I was also the one contributing install_aptget to r-travis ... Yihui Xie spoke up for the "new" Travis approach deploying containers, caching of packages and explicit whitelists. It was in that very (GH-based) discussion that I started to really lose faith in the new Travis approach as they want use to whitelist each and every package. With 6900 and counting at CRAN I fear this simply does not scale. But different approaches are certainly welcome. I posted my 1:03 to 1:15 minutes result. If the "New School" can do it faster, I'd be all ears.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

21 July 2015

Jonathan Dowland: New camera

Sony RX100-III

Earlier in the year I treated myself to a new camera. It's been many years since I bought one, which was a perfectly serviceable Panasonic FS-15 compact, to replace my lost-or-stolen Panasonic TZ3, which I loved. The FS-15 didn't have a "wow" factor and with the advent of smartphones and fantastic smartphone cameras, it rarely left a drawer at home. Last year I upgraded my mobile from an iPhone to a Motorola Moto G, which is a great phone in many respects, but has a really poor camera. I was given a very generous gift voucher when I left my last job and so had the perfect excuse to buy a dedicated camera. I'd been very tempted by a Panasonic CSC camera ever since I read this review of the GF1 years ago, and the GM1 was high on my list, but there were a lot of compromises: no EVF... In the end I picked up a Sony RX 100 Mark 3 which had the right balance of compromises for me. I haven't posted a lot of photos to this site in the past but I hope to do so in future. I've got to make some alterations to the software first. Post-script: Craig Mod, who wrote that GF1 review, wrote another interesting essay a few years later: Cameras, Goodbye, where he discusses whether smartphone cameras are displacing even the top end of the Camera market.

13 June 2015

Craig Small: Linux 4.0 ate my docker images

I have previously written about the gitlab CI runners that use docker. Yesterday I made some changes to procps and pushed them to gitlab which would then start the CI. This morning I checked and it said build failed ok, so that s not terribly unusual. The output from the runner was:

gitlab-ci-multi-runner 0.3.3 (dbaf96f)
Using Docker executor with image csmall/testdebian ...
Pulling docker image csmall/testdebian ...
Build failed with Error: image csmall/testdebian: not found

Hmm, I know I have that image, it just must be the runner so, let s see what images I have:

$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE

Now, I know I have images, I had about 10 or so of them, where did they go? I even looked in the /var/lib/docker directories and can see the json configs, what have you done with my images docker? Storage Drivers The first hint I got from stackexchange where someone lost their AUFS images and needed to load the aufs kernel module. Now I know there are two places or methods where docker stores its images. They are called aufs and devicemapper. There is some debate around which one is better and to be honest with what I do I don t much care, I just want it to work. The version of kernel is significant. It seems the default storage container was AUFS and this requires the aufs.ko kernel module. Linux 4.0 (the version shipped with Debian) does NOT have that module, or at least I couldn t find it. For new images, this isn t a problem. Docker will just create the new images using devicemapper and everyone is happy. The problem is where you have old aufs images, like me. I want those images. Rescue the Images I m not sure if this is the best or most correct way of getting your images, but for me it worked. I got the idea basically from someone who wanted to switch from aufs to devicemapper images for other reasons. You first need to reboot and select at the grub prompt a 3.x kernel that has aufs support. Then when the system comes up, you should see all your images, like this:

$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
csmall/testdebian latest 6979033105a4 5 weeks ago 369.4 MB
gcc 5.1 b063030b23b8 5 weeks ago 1.225 GB
gcc 5.1.0 b063030b23b8 5 weeks ago 1.225 GB
gcc latest b063030b23b8 5 weeks ago 1.225 GB
ruby 2.1 236bf35223e7 6 weeks ago 779.8 MB
ruby 2.1.6 236bf35223e7 6 weeks ago 779.8 MB
debian jessie 41b730702607 6 weeks ago 125.1 MB
debian latest 41b730702607 6 weeks ago 125.1 MB
debian 8 41b730702607 6 weeks ago 125.1 MB
debian 8.0 41b730702607 6 weeks ago 125.1 MB
busybox buildroot-2014.02 8c2e06607696 8 weeks ago 2.433 MB
busybox latest 8c2e06607696 8 weeks ago 2.433 MB

What a relief to see this! Work out what images you need to transfer over. In my case it was just the csmall/testdebian one. You need to save it to a tar file.

$ docker save csmall/testdebian &gt; csmall-testdebian.tar.gz

Once you have all your images you want, reboot back to your 4.x kernel. You then need to load each image back into docker.

$ docker load csmall-testdebian.tar.gz

and then test to see its there

$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
csmall/testdebian latest 6979033105a4 5 weeks ago 368.3 MB

The size of my image was slightly different. I m not too sure why this is the case but assume it is to do with the storage types. My CI builds now run, they re failing because my test program is trying to link with the library (it shouldn t be) but at least its not a docker problem.

8 June 2015

Craig Small: Checking Cloudflare SSL

My website for a while has used CloudFlare as its front-end. It s a rather nice setup and means my real server gets less of a hammering, which is a good thing. A few months ago they enabled a feature called Universal SSL which I have also added to my site. Around the same time, my SSL check scripts started failing for the website, the certificate had expired apparently many many days ago. Something wasn t right. The Problem The problem was simply at first I d get emails saying The SSL certificate for enc.com.au (CN: ) has expired! . I use a program called ssl-cert-check that would check all (web, smtp, imap) of my certificates. It s very easy to forget to renew and this program runs daily and does a simple check. Running the program on the command line gave some more information, but nothing (for me) that really helped:

$ /usr/bin/ssl-cert-check -s enc.com.au -p 443
Host Status Expires Days
----------------------------------------------- ------------ ------------ ----
unable to load certificate
140364897941136:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE
unable to load certificate
139905089558160:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE
unable to load certificate
140017829234320:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE
unable to load certificate
140567473276560:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE
enc.com.au:443 Expired -2457182

So, apparently, there was something wrong with the certificate. The problem was this was CloudFlare who seem to have a good idea on how to handle certificates and all my browsers were happy. ssl-cert-check is a shell script that uses openssl to make the connection, so the next stop was to see what openssl had to say.

$ echo ""   /usr/bin/openssl s_client -connect enc.com.au:443 CONNECTED(00000003)
140115756086928:error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 345 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
---

No peer certificate available. That was the clue I was looking for. Where s my Certificate? CloudFlare Universal SSL uses certificates that have multiple domains in the one certificate. The do this by having one canonical name which is something like sni(numbers).cloudflaressl.com and then multiple Subject Alternative Names (a bit like ServerAlias in apache configurations). This way a single server with a single certificate can serve multiple domains. The way that the client tells the server which website it is looking for is Server Name Indication (SNI). As part of the TLS handshaking the client tells the server I want website www.enc.com.au . The thing is, by default, both openssl s_client and the check script do not use this feature. That was fail the SSL certificate checks were failing. The server was waiting for the client to ask what website it wanted. Modern browsers do this automatically so it just works for them. The Fix For openssl on the command line, there is a flag -servername which does the trick nicely:

$ echo ""   /usr/bin/openssl s_client -connect enc.com.au:443 -servername enc.com.au 
CONNECTED(00000003)
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO ECC Certification Authority
verify error:num=20:unable to get local issuer certificate
---
(lots of good SSL type messages)

That was openssl happy now. We asked the server what website we were interested in with the -servername and got the certificate. The fix for ssl-cert-check is even simpler. Like a lot of things once you know the problem, the solution is not only easy to work out but someone has done it for you already. There is a Debian bug report on this problem with a simple fix from Francois Marier. Just edit the check script and change the line that has:

 TLSSERVERNAME="FALSE"

and change it to true. Then the script is happy too:

$ ssl-cert-check -s enc.com.au -p https
Host Status Expires Days
----------------------------------------------- ------------ ------------ ----
enc.com.au:https Valid Sep 30 2015 114

All working and as expected! This isn t really a CloudFlare problem as such, it is just that s the first place I had seen these sort of SNI certificates being used in something I administer (or more correctly something behind the something).

16 May 2015

Craig Small: Debian, WordPress and Multi-site

For quite some time, the Debian version of WordPress has had a configuration tweak that made it possible to run multiple websites on the same server. This came from a while ago when multi-site wasn t available. While a useful feature, it does make the initial setup of WordPress for simple sites more complicated. I m looking at changing the Debian package slightly so that for a single-site use it Just Works. I have also looked into the way WordPress handles the content, especially themes and plugins, to see if there is a way of updating them through the website itself. This probably won t suit everyone but I think its a better default. The idea will be to setup Debian packages something like this by default and then if you want more fancier stuff its all still there, just not setup. It s not setup at the moment but the default is a little confusing which I hope to change. Multisite The first step was to get my pair of websites into one. So first it was backing up time and then the removal of my config-websitename.php files in /etc/wordpress. I created a single /etc/wordpress/config-default.php file that used a new database. This initial setup worked ok and I had the primary site going reasonably quickly. The second site was a little trickier. The problem is that multisite does things like foo.example.com and bar.example.com while I wanted example.com and somethingelse.com There is a plugin wordpress-mu-domain-mapping that almost sorta-kinda works. While it let me make the second site with a different name, it didn t like aliases, especially if the alias was the first site. Some evil SQL fixed that nicely. UPDATE wp_domain_mapping SET blog_id=1 WHERE id=2 So now I had:

enc.com.au as my primary site
rnms.org as a second site
dropbear.xyz as an alias for my primary site

Files and Permissions We really three separate sets of files in wordpress. These files come from three different sources and are updated using three different ways with a different release cycle. The first is the wordpress code which is shipped in the Debian package. All of this code lives in /usr/share/wordpress and is only changed if you update the Debian package, or you fiddle around with it. It needs to be readable to the webserver but not writable. The config files in /etc/wordpress are in this lot too. Secondly, we have the user generated data. This is things like your pictures that you add to the blog. As they are uploaded through the webserver, it needs to be writable to it. These files are located in /var/lib/wordpress/wp-content/uploads Third, is the plugins and themes. These can either be unzipped and placed into a directory or directly loaded in from the webserver. I used to do the first way but are trying the second. These files are located in /var/lib/wordpress/wp-content Ever tried to update your plugins and get the FTP prompt? This is because the wp-content directory is not writable. I adjusted the permissions and now when a plugin wants to update, I click yes and it magically happens! You will have to reference the /var/lib/wordpress/wp-content subdirectory in two places:

In your /etc/config-default.php: WP_CONTENT_DIR definition
In apache or htaccess: Either a symlink out of /usr/share/wordpress and turn on followsymlinks or an apache Alias and also permit access.

What broke Images did, in a strange way. My media library is empty, but my images are still there. Something in the export and reimport did not work. For me its a minor inconvenience and due to moving from one system to another, but it still is there.

14 May 2015

Craig Small: Hello world!

Welcome to WordPress. This is your first post. Edit or delete it, then start blogging!

11 May 2015

Craig Small: procps using GitLab CI

The procps project for a few years has been hosted at Gitorious. With the announcement that Gitorious has been acquired by GitLab and that all repositories need to move there, procps moved along to GitLab. At first I thought it would just be a like for like thing, but then I noticed that GitLab has this GitLab CI feature and had to try it out. CI here stands for Continuous Integration and is a way of automatically testing your program builds using a bunch of test scripts. procps already has a set of tests, with some a level of coverage that has room for improvement, so it was a good candidate to use for CI. The way GitLab works is they have a central control point that is linked to the git repo and you create runners, which are the systems that actually compile the programs and run the tests. The runners then feed back their results and GitLab CI shows it all in pretty red or green. The first problem was building this runner. Most of the documentation assumes you are building testing for Ruby. procps is built using C with the test scripts in TCL, so there was going to be some need to make some adjustments. I chose to use docker containers so that there was at least some isolation between the runners and the base system. I soon found the docker container I used (ruby:2.6 as suggested by GitLab) didn t have autopoint which mean autogen.sh failed so I had no configure script so no joy there. Now my second problem was I had never used docker before and beyond that it was some sort of container thing so like virtual machines lite, I didn t know much about it. The docker docs are very good and soon I had built my own custom docker image that had all the programs I need to compile and test procps. It s basically a cut-down Debian image with a few things like gettext-bin and dejagnu added in. Docker is not a full system With that all behind me and a few oh I need that too (don t forget you need git) moments we had a working CI runner. This was a Good Thing. You then find that your assumptions for your test cases may not always be correct. This is especially noticeable when testing something like procps which needs to work of the proc filesystem. A good example is, what uses session ID 1? Generally its init or systemd, but in Docker this is what everything runs as. A test case which assumes things about SID=1 will fail, as it did. This probably won t be a problem for testing a lot of normal programs that don t need to dig a deep into the host system as procps does, but it is something to remember. The docker environment looks a lot like a real system from the inside, but there are differences, so the lesson here is to write better tests (or fix the ones that failed, like I did). The runner and polling Now, there needs to be communication between the CI website and the runner, so the runner knows there is something for it to do. The gitlab runner has this setup rather well, except the runner is rather aggressive about its polling, hitting the website every 3 seconds. For someone on some crummy Australian Internet with low speeds and download quotas, this can get expensive in network resources. As there is on average an update once a week or so these seems rather excessive. My fix is pretty manual, actually its totally manual. I stop the daemon, then if I notice there are pending jobs I start the daemon, let it do its thing, then shut it down again. This is certainly not an optimal solution but works for now. I will look into doing something more clever later, possibly with webhooks.

28 April 2015

Craig Small: Backporting and git-buildpackage

For working with Debian packages, one method of maintaining them is to put them in git and use git-buildpackage to build them right out of the git repository. There are a few pitfalls with it, notably around if you forget to import the upstream you get this strange treeish related error which still throws me at first when I see it. Part of maintaining packages is to be able to fix security bugs in older versions of them that are found in stable and even sometimes old stable (jessie and wheezy respectively at the time of writing). At first I used to do this outside git because to me there wasn t a clear way of doing it within it. This is not too satisfactory because it means you lose the benefits of using git in the first place, and for distributions you are more likely to need collaboration with, such as working with the security team or help with backporting. So because I don t think the details are easily found and also I ll probably lose them again and need a central spot to find what I did last time. Building the environment The first step for a new distribution is to create the building environment. You really only need to do this once when there is a new distribution with then possibly some updates from time to time. The command will create the environment in /var/cache/pbuilder/base-DIST.cow/

DIST=wheezy git-pbuilder create

You will find all the files in /var/cache/pbuilder/base-wheezy.cow/ which is then ready to do be used. For more information about this useful tool, look at the git-builder entry on the Debian wiki. Creating the branch The master branch is generally where sid will located. For the other distributions, I use branches. For the initial setup you will need to create the branch and start it off from the right point. For example, the last non-security update for wordpress in jessie was version 4.1+dfsg-1. We then want the jessie branch to start from this point.

git branch jessie debian/4.1+dfsg-1
git checkout jessie

This will then create and put you on the jessie branch. You only need the first line once. You can switch between sid (master branch) and jessie (jessie branch). At this point you have a working place to make all the updates you need to do. Nothing is terribly different from the usual workflow. Building the package Make sure you have checked in all your changes and now it is time to build your package!

git-buildpackage --git-pbuilder --git-dist=jessie --git-debian-branch=jessie

You may need two additional flags:

-sa Use this for the first security update so there is also the source included on the security servers.
git-tag Once you are sure this is the latest changes for this version this will tag in git with the debian tag, such as debian/4.1+dfsg-1+deb8u1

1 April 2015

Benjamin Mako Hill: RomancR: The Future of the Sharing-Your-Bed Economy

Today, Aaron Shaw and I are pleased to announce a new startup. The startup is based around an app we are building called RomancR that will bring the sharing economy directly into your bedrooms and romantic lives. When launched, RomancR will bring the kind of market-driven convenience and efficiency that Uber has brought to ride sharing, and that AirBnB has brought to room sharing, directly into the most frustrating and inefficient domain of our personal lives. RomancR is Uber for romance and sex. Here s how it will work:

Users will view profiles of nearby RomancR users that match any number of user-specified criteria for romantic matches (e.g., sexual orientation, gender, age, etc).
When a user finds a nearby match who they are interested in meeting, they can send a request to meet in person. If they choose, users initiating these requests can attach an optional monetary donation to their request.
When a user receives a request, they can accept or reject the request with a simple swipe to the left or right. Of course, they can take the donation offer into account when making this decision or counter-offer with a request for a higher donation. Larger donations will increase the likelihood of an affirmative answer.
If a user agrees to meet in person, and if the couple then subsequently spends the night together RomancR will measure this automatically by ensuring that the geolocation of both users phones match the same physical space for at least 8 hours the donation will be transferred from the requester to the user who responded affirmatively.
Users will be able to rate each other in ways that are similar to other sharing economy platforms.

Of course, there are many existing applications like Tinder and Grindr that help facilitate romance, dating, and hookups. Unfortunately, each of these still relies on old-fashion intrinsic ways of motivating people to participate in romantic endeavors. The sharing economy has shown us that systems that rely on these non-monetary motivations are ineffective and limiting! For example, many altruistic and socially-driven ride-sharing systems existed on platforms like Craigslist or Ridejoy before Uber. Similarly, volunteer-based communities like Couchsurfing and Hospitality Club existed for many years before AirBnB. None of those older systems took off in the way that their sharing economy counterparts were able to! The reason that Uber and AirBnB exploded where previous efforts stalled is that this new generation of sharing economy startups brings the power of markets to bear on the problems they are trying to solve. Money both encourages more people to participate in providing a service and also makes it socially easier for people to take that service up without feeling like they are socially in debt to the person providing the service for free. The result has been more reliable and effective systems for proving rides and rooms! The reason that the sharing economy works, fundamentally, is that it has nothing to do with sharing at all! Systems that rely on people s social desire to share without money projects like Couchsurfing are relics of the previous century. RomancR, which we plan to launch later this year, will bring the power and efficiency of markets to our romantic lives. You will leave your pitiful dating life where it belongs in the dustbin of history! Go beyond antiquated non-market systems for finding lovers. Why should we rely on people s fickle sense of taste and attractiveness, their complicated ideas of interpersonal compatibility, or their sense of altruism, when we can rely on the power of prices? With RomancR, we won t have to! Note: Thanks to Yochai Benkler whose example of how leaving a $100 bill on the bedside table of a person with whom you spent the night can change the nature of the a romantic interaction inspired the idea for this startup.

Benjamin Mako Hill: RomancR: The Future of the Sharing-Your-Bed Economy

Users will view profiles of nearby RomancR users that match any number of user-specified criteria for romantic matches (e.g., sexual orientation, gender, age, etc).
When a user finds a nearby match who they are interested in meeting, they can send a request to meet in person. If they choose, users initiating these requests can attach an optional monetary donation to their request.
When a user receives a request, they can accept or reject the request with a simple swipe to the left or right. Of course, they can take the donation offer into account when making this decision or counter-offer with a request for a higher donation. Larger donations will increase the likelihood of an affirmative answer.
If a user agrees to meet in person, and if the couple then subsequently spends the night together RomancR will measure this automatically by ensuring that the geolocation of both users phones match the same physical space for at least 8 hours the donation will be transferred from the requester to the user who responded affirmatively.
Users will be able to rate each other in ways that are similar to other sharing economy platforms.

29 January 2015

Craig Small: Juniper Firewalls and IPv6

I found an interesting side-effect of the Juniper firewalls when you introduce IPv6. In hindsight it appears perfectly reasonable but if you are not aware of it in the first place you may have a much more permissive firewall than you thought. My setup is such that my internet address changes every time I connect to an ISP. I have services behind the Juniper that I want to expose onto the Internet, in this case a mailserver. Most of the documentation states to have a reasonably open firewall rule and a nat rule.

[edit security nat]
destination  
    pool mailserver-smtp  
        address 10.1.1.1/32 port 25;
     
 
[edit security policies]
from-zone Internet to-zone Internal  
    policy Mailserver  
        match  
            source-address any;
            destination-address any;
            application smtp;
         
        then  
             permit;

Pretty standard stuff and its documented in plenty of places. We cannot set a destination address because its dynamic, so set it to all. My next step was, ok my mailserver is on IPv4 and IPv6, how do I let the IPv6 connections in? Any means ANY That s where I noticed I had a problem, they could already get in. In fact anyone could get to the mailserver (good) and anything else that had an open SMTP port on my network and used IPv6 (bad). It seems that any destination address means ANY IPv4 or IPv6 address. Both myself and the writers of the documentation hadn t initially thought of what happens when you add IPv6. The solution is to not let any destination, but any IPv4 and the specific mailserver destination. First create an addressbook entry for the IPv6 address of the mailserver.

[edit security address-book global]
address mailserver-ipv6 2001:Db8:1111:2222::100/128;

then adjust the rule

[edit security policies]
from-zone Internet to-zone Internal  
    policy Mailserver  
        match  
            source-address any;
            destination-address [ any-ipv4 mailserver-ipv6];
            application smtp;
         
        then  
             permit;

That way there is access to the mailserver using either IPv4 or IPv6. I m also going to try to see if I adjust the rule so the destination only includes the mailserver address (both IPv4 and IPv6) even though the IPv4 is NATed and see if that works.

Next.

Previous.

Search Results: "craig"

30 December 2015

14 November 2015

4 November 2015

26 October 2015

15 September 2015

8 September 2015

4 September 2015

9 August 2015

8 August 2015

27 July 2015

21 July 2015

13 June 2015

8 June 2015

16 May 2015

14 May 2015

11 May 2015

Related articles Gitorious is being acquired by GitLab and gitorious.org will shut down end of May (about.gitlab.com) Google To Shutdown Google Code, Concedes To GitHub (phoronix.com) GitLab Acquires Gitorious (developers.slashdot.org)

28 April 2015

1 April 2015

29 January 2015

Related articles

Gitorious is being acquired by GitLab and gitorious.org will shut down end of May (about.gitlab.com)

Google To Shutdown Google Code, Concedes To GitHub (phoronix.com)

GitLab Acquires Gitorious (developers.slashdot.org)